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FOREWORD 


No  om  questions  the  need  for  quality  training  program.  However,  it 
is  very  appropriate  that  ve  attempt  to  get  the  aoet  value  out  of  our 
training  "dollar."  Ae  a  natter  of  fact,  this  phase  of  our  training 
effort  la  long  overdue. 

Hera  le  a  report  of  an  attempt  to  develop  a  net hod  of  training  eval¬ 
uation  that  is  closely  identified  with  ope rat lone 1  requlreuents  and 
active  supervisory  responelblllt  It  was  designed  by  our  Personnel 
Research  Staff.  It  can  be  implemented  at  any  organisational  level. 

This  report  contains  inf onset ion  that  raises  pointed  questions.  Care¬ 
ful  study  of  quest lane  like  these  eon  show  us  how  to  overcem  short¬ 
comings  in  our  training  effort,  and  how  our  total  training  program  can 
sMke  Its  met  effective  contribution. 

In  addition  to  Problem  Analysis  and  Decision-Making  training,  we  are 
already  applying  this  method  to  the  SID  and  SIMM  program.  All  of 
you  who  have  a  share  in  training  efforts  will,  I  am  sure,  see  other 
ways  in  which  this  method  can  be  put  to  work  to  point  up  more  ques¬ 
tions  amd  develop  more  information  that  can  be  of  value  to  you. 
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SUMMARY  AND  HIGHLIGHTS 


The  purpose  of  this  research  was  to  develop  a  method  which  could  terse 
as  a  prototype  for  evaluating  managerial  traini-^.  In  addition  to  exemplify¬ 
ing  the  general  rules  underlying  all  training  evaluation,  the  procedure  was 
designed  to  test  the  effectiveness  of  the  training  effort  against  specific 
requirements  of  a  sound  training  program.  These  requirements  include  aware¬ 
ness  by  participants  and  their  supervisors  of  the  specific  objectives  of  the 
proposed  training,  encouraging  continuing  attention  to  employee  developamnt 
needs,  the  integration  of  training  with  operations,  and  adequate  follow-up 
by  supervisors  as  trail  as  by  training  staffs.  ' 

Two  sets  of  two  questionnaires  each  were  constructed.  One  set  was  for 
participants;  the  other  for  supervisors.  Each  sat  Included  a  questionnaire 
to  be  administered  before  training,  and  another  for  use  Immediately  after 
training.  The  questionnaires  were  adadnlstared  in  connection  with  the 
Kepner-Tregoe  course  in  Management  Problem  Analysis  and  Decision  Making. 

Of  a  t.otal  of  54  participants  in  five  separate  sessions  of  the  course,  53 
returned  completed  questionnaires.  Returns  from  supervisors  were  approxi¬ 
mately  751.  Responses  of  both  groups  were  content-analysed  and  classified 
according  to  level  or  quality  of  content. 

RESPONSES  TO  PRE-TRAIHING  QUESTIONS 

Most  participants  and  supervisors  had  more  or  less  clearly  expressed 
ideas  as  to  the  objectives  of  the  course,  hut  only  191  of  the  super¬ 
visors  and  261  of  the  participants  identified  these  objectives  with 
the  work  of  the  participant  or  his  value  as  an  employed. 

131  of  the  participants  and  261  of  the  supervisors  gave  responses 
that  reflected  no  awareness  af  career  plans  or  career  development. 

261  of  participants  and  691  of  supervisors  were  Ale  to  mention 
specific  problems  which  t'Sey  hoped  would  ha  handled  mors  affectively 
by  the  participant  after  training.  961  and  161,  respectively,  were 
unable  to  point  to  say  specific  problem  or  problem  area  to  which  they 
thought  the  training  might  ha  applicable. 

In  approximately  901  of  the  cases  no  discussion  took  place  between 
participant  and  supervisor  prior  to  training. 

About  301  of  both  groups  cited  specific  incidents  of  jab  performance 
that  pointed  up  the  need  for  this  kind  of  training.  9§1  of  partici¬ 
pants  and  471  of  supervisors  failed  to  present  ay  specific  evidence 
of  training  needs. 

731  of  supervisors  and  621  of  participants  considered  the  advance 
information  concerning  the  course  adequate.  271  sad  381,  respectively, 
felt  that  it  was  insufficient  or  non-existent. 
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Approximately  25%  of  the  responses  explicitly  mentioned  the  parti¬ 
cipant  es  taking  part  in  the  declelon  that  he  attend  the  courae. 

11%  of  the  participants  were  unable  to  say  who  had  taken  part  in 
the  decision. 

RESPONSES  TO  POST -TEA IN INC  QUESTIONS 

21%  of  the  supervisors  and  58%  of  the  participants  reported  changes 
in  their  ideas  concerning  the  objectives  of  the  course. 

The  percentage  of  participants  who  were  able  to  euggest  specific 
applications  in  their  regular  Jobe  Increased  from  26%  before  train¬ 
ing  to  46%  after  training.  The  percentage  who  were  unable  to  visu¬ 
alise  any  useful  applications  of  the  course  dropped  from  34%  to  14%. 

Of  42  participants  for  whom  complete  data  were  available,  50%  were 
able  both  to  specify  applications  of  the  training  and  to  cite  situa¬ 
tions  which  they  would  have  handled  differently  if  they  had  already 
taken  the  course.  7%  were  unable  co  indicate  either  applications  or 
situations  that  they  would  have  handled  differently. 

Most  participants  and  supervisors  indicated  same  plans  for  mutual 
discussion  and  follow-up  regarding  the  results  of  the  training. 

QUESTIONS 

Is  there  sufficient  cannunlcat ion  between  training  staff  and  line 
supervisors  concerning  the  content  sad  objectives  of  available 
courses? 

Is  the  supervisor  usually  provided  with  enough  information  to  enable 
him  to  make  the  best  choices  among  courses  and  among  potential 
tralaeeef 

If  s  supervisor  Is  unable  to  document  specific  training  needs,  or 
to  specify  ways  in  which  hs  aspects  employees  to  apply  the  proposed 
training,  on  what  baa is  does  hs  select  trainees? 

Is  too  heavy  reliance  placed  upon  the  M quota"  method  of  assignment? 
Cen  methods  of  selecting  participants  be  improved? 

Is  the  mere  hops  that  an  eaployee  will  be  "helped"  by  a  course,  about 
which  the  supervisor  knows  little  more  than  its  title,  enough  justi¬ 
fication  for  the  cost  of  the  training? 

Do  supervisors  sufficiently  discuss  the  proposed  training  with  their 
employees? 
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Is  ths  awareness  of  ths  various  responsibilities  for  aanagenent 
development,  hold  and/or  aharad  In  each  organisation,  as  clear  as 
it  night  be? 

Would  the  use  of  thia  kind  of  evaluation  procedure  on  a  systematic 
and  continuing  basis  result  in  a  "tightening  up"  of  the  training 
effort  all  along  the  line?  Would  such  an  outcome  be  desirable? 
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A  PATH  TO  MANAGEMENT  DEVELOP* NT  AND  TO 
THE  MEASUREMENT  OF  ITS  GROWTH1 


Z.  Introduction 


There  Is  no  denying  the  fact  that  training,  particularly  management 
training.  Is  most  difficult  to  appraise.  Tangible  effects  directly  attri¬ 
butable  to  such  training  are  quite  elusive,  and  they  may  occur  at  a  time 
relatively  remote  from  the  course  of  Instruction. 

But  lack  of  reasonably  objective  evidence  makes  those  of  us  who 
sponsor  and  support  management  training  vulnerable  to  our  critics,  who 
like  to  see  evidence  in  physical  or  fiscal  terms.  Also,  among  those 
responsible  for  training,  the  deficiency  of  solid  data  makes  some  unduly 
susceptible  to  enthusiasm,  while  others  hesitate  to  launch  or  subscribe 
to  potentially  good  programs  for  fear  that  they  might  be  unable  to  justify 
their  judgment  adequately. 

The  measure  of  effective  training  is  change— change  in  performance 
that  contributes  to  more  effective  mission  accomplishment  for  the  organi¬ 
sation  sponsoring  the  training.  First  and  foremost,  of  course,  we  record 
changes  in  the  man.  But  this  is  not  enough,  if  we  are  to  appraise  the 
success  of  the  training  effort  in  terms  of  the  total  outcome.  In  our 
intense  concentration  upon  "employee  development"  we  often  lose  sight  of 
the  fact  that  employee  development  is  an  abstraction  unless  it  is  care¬ 
fully  and  systeswticaily  integrated  with  the  overall  requirements  of 
"management  development."  Hence,  the  effects  that  we  look  for  should  be 
reflected  not  only  in  changes  in  the  man,  but  also  in  hie  supervisor,  in 
higher  management,  and  in  the  total  work  situation.  The  training  of 
managers  is  but  one  phase  of  management  development. 

Are  the  supervisor  and  the  management  ready  for  the  "changed  man?" 
This  question  is  not  often  asked  prior  to  assigning  an  individual  to 
training.  It  may  seem  like  a  silly  question,  because  it  is  "self-evident" 
that  an  employee  would  n>rt  be  sent  to  a  course  unless  it  was  expected  that 
he  would  leern  something  of  benefit  to  the  organisation.  However,  some 
interesting  research  has  dlsclosad  that  a  man  may  not  be  able  to  use  the 
knowledge  gained  because  it  involves  changes  in  the  man  or  in  the  situa¬ 
tion  that  his  associates  are  not  used  to,  and  deliberately  or  unwittingly 
are  unable  to  accept.  In  some  cases  it  has  even  been  shown  that  the  man 
is  more  successful  in  applying  what  he  has  learned  if,  after  training,  he 
gc«s  to  a  different  shop  or  office  than  tha  one  he  started  out  from.  In 


^is  report  describes  the  first  phase  of  a  research  project  under¬ 
taken  at  the  request  of  tha  Bmployee  Development  Division  of  the  Office 
of  Personnel. 
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the  new  piece,  Che  established  conception*  ee  to  how  Joe  is  suppose  to 
ect  end  how  he  is  to  function  ere  not  present  to  hobble  hie. 

It  holds  from  this  point  of  view  thet  benefits  from  treining  ere  e 
function  not  only  of  the  nature  end  degree  of  participation  and  experi¬ 
ence  of  the  employee -trainee,  but  also  of  the  role  played  by  supervisors 
and  Managers ,  as  well  as  the  interaction  among  all  parties  Involved  in 
planning  for  training  and  for  its  application. 

This  was  the  orientation  with  which  we  approached  the  problem  of 
developing  a  prototype  system  for  more  effectively  managing  and  evalue*  i 
managerial  training.  And  from  this  orientation  derived  the  statements  .• 
specific  objectives  of  this  research  effort  (an  action  research  program 
part),  end  of  the  outcomes  of  training  expected  to  serve  as  a  basis  for 
judging  the  degree  of  its  success. 
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II.  What  He  Looked  For 


General  Requirements 

Five  general  rules  guided  the  present  effort.  These  rules  are  best 
exemplified  by  questions  which  serve  to  define  the  requirements  that  the 
evaluation  must  satisfy. 

1.  What  specific  things  Is  the  trainee  expected  to  do,  or  to  do 
differently  as  a  result  of  this  training  experience?  The  effectiveness 
of  any  training  program  can  only  be  assessed  in  terms  of  the  specific 
objectives  of  that  program.  Furthermore,  statements  of  objectives  must 
always  point  toward  observable  behaviors  or  behavior  changes.  If  evalua¬ 
tion  is  to  be  possible. 

2.  How  did  the  individual  act  before  he  was  trained?  How  did  he 
act  after  training?  Since  the  effect*  of  training  appear  as  changes  in 
behavior,  it  is  necessary  to  have  a  "bench  stark"  to  know  vhere  we  started 
from. 


3.  Are  the  long-term  effects  the  same  as  the  more  ismediate  changes 
in  behavior?  Often  the  two  pictures  will  lead  to  quite  different  con¬ 
clusions. 


4.  What,  precisely,  do  we  wish  to  evaluate--the  content  of  the  pro¬ 
gram,  the  method  of  teaching  It,  the  ability  of  the  ii  .  rue tors,  or  what? 
For  present  purposes  it  seemed  most  desirable  to  appraise  the  combined 
impact  of  all  such  factors,  rather  than  try  to  identify  the  specific 
effects  of  each  separately. 

5.  What  use  is  to  be  made  of  the  evaluation  process?  A  procedure 
like  the  one  to  be  described  has  many  possible  applications.  It  can 
provide  an  objective  appraisal  of  the  effectiveness  of  a  particular  course. 
Used  systematically  and  on  a  continuing  basis,  it  can  be  a  means  of  tracing 
changes  in  the  effectiveness  of  the  training  program  over  a  period  of  time. 
By  pinpointing  specific  weaknesses  either  in  a  training  program  or  in  its 
administration,  it  can  lead  eventually  to  better  results  from  the  training 
effort.  The  understanding  that  a  'dloy-up  is  expected  to  be  a  regular 
part  of  the  training  process  should  stimulate  participants  and  their  super¬ 
visors  to  think  more  carefully  about  any  program  that  is  under  considera¬ 
tion,  and  more  consistently  about  training  needs  and  employee  development 
as  a  continuing  responsibility.  More  discriminating  selections  both  of 
courses  and  participants  would  represent  a  substantial  gain  in  Itself. 

Specific  Requirements 

What  are  the  desirable  features  of  a  sound  training  program,  the 
things  to  look  for  in  planning  an  evaluation?  The  following  list  of 
specifics  is  not  necessarily  exhaustive;  however,  it  covers  the  main 
points  which  determined  our  approach  tq  the  task: 
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1.  Both  participant!  and  supervisors  should  display  sons  advance 
knowledge  of  the  apeciflc  objective!  of  the  training.  This  should  be 
cOwbined  with  evidence  of 

2.  Knowledge  of  the  employee's  individual  training  needs,  and 

3.  Perception  of  the  role  of  this  particular  training  in  relation 
to  his  career  development.  At  the  sane  tine, 

4.  Training  ahould  be  integrated  with  operations--supervlsora  and 
particlpante  alike  should  be  able  to  indicate  definite  ways  in  which  the 
employee  is  expected  to  nee  the  training  in  his  work.  This  Implies 

5.  Comaunlcation  between,  supervisor  and  employee,  indicating  that 
both  are  involved  in  planning  the  training  effort.  It  also  calls  for 

6.  Adequate  feedback  to  the  supervisor,  as  well  as  to  agency  and 
departmental  training  staffs. 
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III.  WhAT  WE  DIO 


An  essential  requirement  of  the  prototype  was  that  it  must  be  appli¬ 
cable  to  many  afferent  training  programs.  This  requirement  obviously 
eliminates  the  possibility  of  laying  down  any  single  set  of  performance 
criteria  againet  which  all  programs  can  be  appraised.  Actually,  this  is 
not  a  serious  handicap  in  view  of  what  we  were  looking  for.  The  charac¬ 
teristics  of  a  good  program,  as  Just  outlined  are  common  to  all  training, 
as  &re  the  requirements  of  feedback  to  supervisors  and  management.  At 
the  same  time  we  may  assume  that  both  employee  and  supervisor  can  recog¬ 
nize  effective  and  ineffective  job  performance,  that  they  can  translate 
the  latter  into  expressions  of  training  need  and,  finally,  that  they  can 
recognize  changes  in  performance  when  or  if  they  occur. 

To  try  out  the  technique,  a  five-day  course  in  "Management  Problem 
Analysis  and  Decision-Making"  was  selected.*  This  course  is  offered  on  a 
contract  basis  to  government  agencies  and  industrial  firms  by  Kepner,  Tregoe 
and  Associates,  Inc.,  of  Princeton,  New  Jersey.  During  1962  a  number  of 
sessions  were  held  for  USDA  employees  in  GS  grades  12  to  15.  Conducted  as 
a  workshop  and  seminar,  the  course  teaches  specific  concepts  and  methods 
which  appear  to  be  directly  applicable  in  the  trainee's  daily  work. 

Two  sets  of  two  questionnaires  each  were  constructed  (Appendix  A).  One 
set  was  for  trainees;  the  other  for  their  supervisors.  Each  set  Included 
one  questionnaire  to  be  administered  prior  to  training;  and  another,  Imme¬ 
diately  after  training.  A  third  pair  of  questionnaires  will  be  used  six 
months  or  more  following  completion  of  the  training.  This  final  follow-up 
will  be  described  in  a  later  report. 

"Before"  end  "After"  quest lonna ires  were  given  to  all  participants  in 
five  separate  sessions  of  the  Kspner-Tregoe  course  held  between  October  15 
and  December  14,  1962.  The  total  group  consisted  of  54  individuals,  of 
whom  53  returned  the  information  requested.  Exhibits  1  and  2  show  the  dis¬ 
tributions  of  these  participants  by  USDA  agencies  and  by  grade  levels, 
respective ly. 

Pre-training  questionnaires  wars  mallsd  to  ths  supervisors  of  all  par¬ 
ticipants.  Post-training  questionnaires  wars  sent  to  supervisors  of  the  44 
participants  in  the  last  four  suasions  of  the  course— the  first  session  was 
not  represented  because  this  questionnaire  was  not  developed  in  tins.  A 
return  of  approximately  75%  was  obtained  in  both  cases. 

The  responses  to  each  question  were  sorted  into  categories  according 
to  the  level  of  quality  of  the  content.  No  more  then  three  categories  were 


2See  Kepner,  Charles  H.  and  Tregoe,  Benjamin  B.,  Developing  decision 
makers,  Harvard  Business  Review.  1960,  36,  115-124,  for  a  brief  account  of 
the  course . 
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used  in  any  caaa.  Each  category  vaa  scored  2,  1,  or  0;  the  higher  number 
always  indicating  the  "better"  or  preferred  answer,  as  will  be  illustrate’* 
shortly.  When  the  question  was  of  the  "yes-no"  type,  only  two  were  needed. 


Exhibit  1 

Distribution  of  Participants  by  US DA  Agencies 


Agency 

Humber  of 
Participants 

Forest  Service 

36 

Agricultural  Marketing  Service 
Agricultural  Stabilisation  and 

5 

Conservation  Service 

3 

Agricultural  Baeearch  Service 

4 

Farmer  Cooperative  Service 

1 

Motional  Agricultural  Library 

1 

Soil  Conservation  Service 

1 

Total 

53 

The  cetegerlee  used  In  classifying  the  answers  to  each  question  were 
determined  primarily  by  content  analysis.  It  was  relatively  easy  in  this 
way  to  group  a  given  set  of  answers  under  two  or  three  broad  themes  which 
were  relevant  to  tbs  objectives  ef  the  study.  The  classification  schemes 
thus  developed  wore  tooted  for / objectivity  by  checking  the  agreement 
between  sorters  who  worked  independent ly>  When  members  of  the  Personnel 
Kesearch  Staff  sorted  random  samp lss  of  answers  In  this  way,  agreement 
ranged  between  101  end  1001  from  one  question  tv  nother. 


Exhibit  2 

Distribution  of  Participants  by  Grads  Level 


GO  Level 

Number  of 
Participants 

IS 

1 

u 

27 

IS 

IS 

IS 

9 

Total 

53 

t 
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IV.  What  V*  Found 


Pre -Training 

The  pre-training  questionnaires  for  both  participants  and  super¬ 
visors  contain  seven  main  items,  plus  an  eighth  which  invitee  "other 
consents."  Since  the  two  sets  of  questions  are  generally  parallel, 
content  analyses  yield  the  seme  categories  for  both  sets.  These 
categories  will  now  be  presented  for  each  question  in  turn,  together 
with  a  summary  of  their  frequency  of  occurrence  among  participants 
and  supervisors. 
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Quest lem  Prs-1:  WHAT  DO  YOU  THUS  ARE  THE  SPECIFIC  OBJECTIVES  OF  THE 
COURSE T 


Levi  2  -  Responses  conteinlng  explicit  reference  to  the  partlcipeat  In 
terns  of  epplylng  the  training  in  his  present  or  future  work,  or  lnprovin 
his  value  to  the  organisation. 

"To  increase  my  ability  and  efficiency  as  an  adnlnlstrator  by  train! 
in  Methods  cf  problcn  solving  and  decision-asking." 

"To  give  this  participant  knowledge  which,  if  applied,  will  enable  •/ 
to  nake  a  greater  percentage  of  right  decisions,  thus  lap  roving  hi* 
perfomence  and  incraaslng  his  value  to  his  eaployers." 

Level  -  General  statenent  without  explicit  reference  to  participant'^ 
present  work  of  future  value  to  the  organisation. 

"Help  imagers  do  a  better  Job  of  solving  problene  and  Making  declsi> 

"To  iapdbv  the  skill  of  analysing  problene  and  asking  decisions." 

Level  0  -  General  statenent  without  explicit  reference  to  either  problem 
analysis  or  declsion-naklng. 

"The  course  will  present  a  systeaatlc  asthod  of  reasoning." 

"To  assist  participants  la  understanding  tbs  slansnts  they  use  in  met 
lag  everyday  work  situations." 
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Exhibit  3 

PercenMge  Distribution*  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-1 


PARTICIPANTS  SUPERVISORS 


Exhibit  3  shows  the  percentages  of  participant*  and  of  aupervisora 
whose  answers  to  Question  Pre-1  fell  in  each  of  the  three  categories. 

The  two  groups  were  wuch  alike  in  their  understanding  of  the  objec¬ 
tives  of  this  particular  course.  As  maker  who  had  no  idea  of  the 
objectives  was  negligible.  Most  had  a  nor*  or  less  clearly  expressed 
idea  in  their  answers  (Level  1),  hut  only  19X  of  the  supervisors  and 
26X  of  tbs  participants  explicitly  identified  these  objectives  with  the 
participant's  actual  work  or  his  value  as  an  saployee.  Others,  no 
doubt,  took  this  Identification  for  granted;  nevertheless  it  would  have 
been  suggestive  of  aore  careful  thought  and  planning  if  they  had  ex¬ 
pressed  the  idea  in  so  aany  words. 
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QUMftion  Pr*-2:  HOW  DOBS  THIS  SEEM  TO  FIT  INTO  YOUR  OWN  CAREER  PLANS? 
(Participants) 

HOW  DOBS  THIS  FIT  INTO  YOUR  AGENCY  TRAINING  AND  DBVELO 
KENT  PROGRAM  FOR  THIS  PARTICIPANT?  (Supervisors) 


Level  t  -  Response  contains  sons  allusion  to  long-range  plans,  e.g. , 
continuing  in  present  career,  changing  career  field,  or  advancing 
within  career.  (Participants) 

Responsa  contains  explicit  allusion  tc  potential  of  parties  *' 
eft  bar  in  his  present  position  or  in  future  assignments.  (Supervisor.) 

" In  my  position  the  training  should  help  toward  better  redeeming 
my  responsibilities  as  a  manager,  thereby  enhancing  opportunity 
for  continuing  up  the  career  ladder." 

"This  participant  apparently  has  a  high  potential  as  a  research 
administrator.  His  ability  will  be  needed  in  expanding  USDA 
program.  This  training  was  assisted  to  increase  his  ability  to 
handle  his  present  and  future  assignment* ." 

level  1  -  Response  refers  only  to  immediate  demands  of  present  Job. 
(Participants) 

Responsa  indicates  that  course  is  considered  part  of  agency's 
regular  training  program,  but  contains  no  specific  reference  to  parti¬ 
cipant's  potential  or  his  career  plans.  (Supervisors) 

"I  am  a  branch  chief  with  nation-wide  programs  calling  for  frequent 
decisions.  X  aspect  this  course  to  be  useful  to  as  in  running 

these  programs. M 

"Our  Division  has  a  formalised  training  program  which  calls  for 
this  employee  to  taka  courses  which  will  provide  across  the  board 
management  training.  This  particular  epurse  will  fill  a  part  of 
the  requirements  of  this  program." 

Level  £  -  Mo  specific  tie-in  either  with  present  work  or  with  future 
plans.  (Participants) 

General  statement  as  to  effectiveness  of  and  need  for  such 
training,  but  no  specific  description  of  its  role  in  participant's 
career  development.  (Supervisors) 

"Fits  in  well." 

"A  supervisor's  deputy  should  analyse  problems  and  sake  decisions. 
Ability  to  do  so  is  essential  to  the  job." 


Exhibit  4 


Percentage  Diatr lout  ions  of  Responses  by  Participants 
and  S  ipervisors  to  Ouestlon  Pre-2 

100 1 - 1 


The  percentage  distributions  of  answers  to  Ouestlon  Pre-2  are 
shown  in  Exhibit  4. 

It  is  interesting  that  45%  of  the  participants,  but  only  19%  of 
the  supervisors  responded  at  Level  2.  This  night  suggest  that,  as  a 
group,  eaployees  have  a  clearer  picture  of  their  career  plans  and  ot 
the  role  of  this  training  In  relation  to  those  career  plans  than  do 
their  supervisors.  Supervisors  as  a  group  seen  to  have  had  a  tendency 
to  think  of  the  course  nora  In  tens  of  fornelly  prescribed  training 
prograns  than  In  terns  of  Individual  career  plans  (55%  of  answers 
scored  at  Level  1).  The  apparent  difference,  however,  nay  be  an  arti¬ 
fact,  since  the  supervisors'  question  Is  not  explicitly  career-oriented. 
In  any  event,  It  nay  be  disconcerting  to  note  that  13%  of  the  partici¬ 
pants  and  26%  of  the  supervisors  gave  answers  that  reflected  no  aware¬ 
ness  of  career  plans  or  career  developnsnt.  These  differences  between 
supervisors  and  participants  are  highly  significant  statistically. 

That  Is,  they  are  too  large  to  be  attributed  to  chance. 3 


“A  two-tailed  sign  test  was  significant  at  the  .04%  level  of 
confidence. 
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Question  Pre-3:  ON  THE  BASIS  OF  INFORMATION  AVAILABLE  TO  YOU  NON,  IN 

WHAT  PARTICULAR  NAYS  DO  YOU  EXPECT /EXPECT  THIS  EMPLOYEE 
TO  APPLY  THIS  TRAINING  IN  YOUR/HIS  FUTURE  WORK?  GIVE 
EXAMPIZS. 


Level  2  -  Response  mentions  specific  task(s)  or  problem(s) . 

"I  expect  the  employee  to  use  the  material  to  determine  what  is  the 
cheapest  and  siost  effective  technique  to  use  in  controlling  a  sped 
fic  fire." 

"Help  decide  between  (a)  establishment  of  a  study  to  limit  of  pre?ir 
capabilities  or  work  to  get  tupport  of  isore  adequate  study;  (b)  bet 
up  program  of  research  in  our  awn  organisation  or  contract  with 
another  agency." 

Level  _1  -  Reference  to  general  task  or  problem  area,  but  without  specif* 
examples. 

"In  developing  regulations  and  Instructions  for  new  programs  for 
which  I  have  been  given  responsibility;  in  analysing  questions  and 
problems  presented  by  field  offices  with  respect  to  these  new  pro¬ 
grams  and  also  programs  which  are  my  primary  responsibility." 

"This  employee  is  in  effect  our  Public  Relations  expert.  Ve  look 
to  him  for  guidance  as  to  how  our  agency  can  best  carry  out  its 
program  with  public  acceptance  and  support.  This  Involves  careful 
analysis  of  smny  factors,  often  not  lined lately  apparent  as  impor¬ 
tant-how  this  group  or  that  group  will  react  and  why;  or  can 
management  attain  the  same  objectives  but  in  a  way  that  is  more 
acceptable  to  the  affected  publlcT" 

Level  0  -  General  statements  without  reference  to  any  actual  task  or 
prdSTan  area. 

"Making  decisions  is  pert  of  the  delly  Job.  Training  in  good 
decision-making  will  be  used  everyday." 

"This  employee  will  be  faced  with  problems  national  in  scope  and 
extending  across  all  commodity  lines  on  a  day  to  day  basis.  If 
it  doesn't  help  him  it  is  no  good  or  mislabeled." 


V  «% 
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Exhibit  5 

Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Ouestion  Pre-3 

100 


z 

UJ 

u 

a t 

liJ 
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Although  Exhibit  S  shows  soae  apparent  differences  between  super¬ 
visors  and  participants  in  their  responses  to  Ouestion  Pre-3,  these 
differences  are  not  statistically  significant.  The  aost  that  we  can 
say  here,  therefore,  is  that  while  fairly  sisable  lumbers  of  both  par¬ 
ticipants  (26%)  and  supervisors  (431)  were  able  to  sent ion  specific 
p  rob  leas  which  they  hoped  wc  ild  be  handled  aore  effectively  afte.  coa- 
p let ion  of  the  course  (Level  2),  considerable  ousters— 34%  and  24%-- 
were  unable  to  point  to  any  specific  problea  or  any  problea  ares  to 
which  they  thought  the  training  night  be  applicable.  Apparently,  they 
hoped  that  the  participant  alght  benefit  froa  the  course  in  sons  way. 


0  1  2 
LEVEL 

PARTICIPANTS 


SUPERVISORS 
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Question  Pre-4:  HAVE  YOU  DISCUSSED  POINTS  1,  2,  AND  3  WITH  YOUR 

SUPERVISOR/THIS  EMPLOYEE?  IF  YES,  DO  YOU  FEEL  THAT 
HE  SHARES  THE  SAME  EXPECTATIONS  AS  YOU  DO?  IF  NOT, 
HON  DO  YOUR  EXPECTATIONS  DIFFER  AS  YOU  SEE  IT? 


Level  2  -  "Yes’’  to  both  questions. 

Level  ^  -  "No"  to  first  question;  "Yes"  to  second. 

Level  0  -  "No"  to  first  question,  no  answer  or  "don't  know"  to  second. 


Exhibit  6 

Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-4 

100  I - 1 


As  night  be  expected.  Exhibit  6  choirs  close  agreensnt  between  par¬ 
ticipants  sad  supervisors.  Practically  no  one  on  either  side  of  the 
fence  reported  feeling  that  his  expectations  were  not  shared.  Perhaps, 
the  aost  noteworthy  finding  brought  out  by  this  question  is  that  In 
approxlaately  half  the  esses,  before  the  training,  no  discussion  took 
place  between  supervisor  and  participant  (only  47%  of  participants  and 
55%  of  supervisors  gave  Level  2  responses). 
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Question  Pre-5:  LOOKING  BACK  OVER  TOUR/HIS  WORK  DURING  THE  PAST  SIX 
MONTHS  OR  SO,  HAS  ANYTHING  HAPPENED  TO  BRING  HOW  TO 
YOU  THE  NEED  FOR  THIS  KIND  OF  TRAINING?  GIVE  ACTUAL 
EXAMPLES. 


Level  2  -  Response  contains  examplc(s)  with  some  specification  of  problem(s). 

"Difficulty  in  hiring  key  man  in  one  discipline  forced  us  into  con* 
trnct  with  a  university.  Had  I  been  willing  and  able  to  read  hand¬ 
writing  on  wall  sooner,  we  could  have  made  a  better  'marriage'  and 
goi.ten  work  underway  sooner." 

"He  sent  a  subordinate  on  a  mission,  the  results  of  which  were  not 
thought  through,  and  the  mission  had  to  be  called  off.  This  could 
have  been  avoided  by  thorough  consideration  of  the  consequences." 

Level  ji  -  Task  or  problem  area  indicated,  but  specific  example(s)  not  given. 

"For  almost  a  year  I  have  been  evaluating  the  magnitude,  cost  and 
direction  being  taken  In  service-wide  programs.  To  prescribe  policy 
and  procedure  is  the  next  step,  then  evaluate  progress." 

"The  problems  of  federal  land  use  and  administration  with  which  this 
office  deals,  are  becoming  increasingly  complicated.  Values  are  con¬ 
tinually  enhanced,  competition  for  lands  more  intense.  This  brings 
to  the  forefront  the  need  for  increasing  technical  competency  and 
accurate  thinking  on  specific  problems." 

Level  0  -  General  answer— no  specific  task  or  problem  area  indicated. 

"Nothing  specific.  In  general,  tbs  need  to  keep  ahead  of  a  rapidly 
increasing  activity  with  many  growing  problems." 

"Neve* ' t  been  associated  long  enough  to  know  his  need  for  this  kind 
of  training— other  than  Z  know  it  will  prove  helpful." 
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Exhibit  7 

Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-5 


100 


z 


Xu  with  Questions  1,  3,  and  4.  supervisors  and  participants  did 
not  differ  significantly  In  the  distributions  of  their  replies  to 
Question  Pre-5.  Exhibit  7  shows  that  approximately  30X  of  both  groups 
cited  specific  Incidents  of  job  performance  that  pointed  up  a  need  for 
training  In  problem  analysis  and  declslon-msklng.  Another  25X  to  34X 
were  aware  of  geoaral  task  areas  la  which  this  need  appeared.  Once 
again,  however,  u  large  proport Ion- -34X  of  the  participants  and  47X 
rf  the  super/ is ors— failed  to  present  any  specific  evidence  of  train¬ 
ing  needs. 
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COMPARISON  OP  QUESTIONS  PRE-3  AND  PRE-5 


Since  Questions  Pre-3  end  Pre-5  are  somewhat  related,  It  ie  lnforma- 
tive  to  compare  the  distribution*  of  reaponaes.  The  participants  were 
quite  consistent,  as  Exhibit  8  shows.  The  supervisors,  ou  the  other  hand, 
showed  a  higher  proportion  of  Level  2  answers  on  Question  Pre-3,  and  a 
higher  proportion  of  tero  answers  on  Question  Pre-5.  These  difference  , 
were  significant  statistically. ^  This  would  seen  to  indicate  that  Super1* 
visors  generally  are  more  ready  to  point  to  job  areas  where  they  hope  thdt 
the  training  will  be  applied  then  to  document  specific  needs  for  the 
training. 


Exhibit  8 

Percentage  Distribution  of  Responses 
on  Creations  Pre-3  and  5 


Pre- 


Partlclnanta 


gupervleore  _ 
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34 

40 
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23 
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A  two-tailed  Sign  test  was  significant  at  the  3%  leva',  c*  on  f  Ida  ace. 
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Question  Pre-6:  IS  THERE  ANY  OTHER  ADVANCE  INFORMATION  THAT  YOU  THINK 
SHOULD  HAVE  BEEN  PROVIDED  ABOUT  THE  COURSE? 


Laval  _1  -  Any  answer  that  indicates  no  wish  for  additional  information. 

Laval  0  -  Any  answer  that  indicates  need  for  further  information. 

Seventy-three  percent  of  supervisors  and  62%  of  participants  con¬ 
sidered  the  advance  information  adequate;  27%  and  38%,  respectively, 
felt  that  it  was  insufficient.  In  this  connection  it  might  be  pointed 
out  that  there  ia  more  advance  information  available  concerning  the 
Kepner-Tregoe  course  than  there  is  for  many  training  programs  in  which 
government  agencies  participate.  Evidently,  then,  tl.ere  must  be  some 
breakdowns  in  communication  along  the  line.  This  is  best  illustrated 
by  quoting  a  few  answers  to  Question  Pre-6. 

Participants:  "I  would  like  to  have  had  clear  understanding  that  the 
course  will  benefit  staff  people.  Decision-making  can 
be  construed  as  applying  primarily  or  even  exclusively 
to  line  administrators." 

"I  really  received  no  advance  information  that  I  can  re¬ 
call.  Should  have  gotten  summary  of  previous  trainees' 
impressions  plus  more  on  actual  content  of  course. 
Chapter  I  of  course,  sent  to  me  in  advance,  gave  Impres¬ 
sion  course  might  be  too  'business  oriented' for  my  needs, 
and  I  almost  withdraw." 

"Course  outlines  explained  by  training  officers  periodi¬ 
cally  (to  group)  would  help  one  select  training  oppor¬ 
tunities  to  priority  better.  To  change  mental  gears 
quickly  when  a  bulletin  comas  by  Is  fraught  with  error, 
disinterest  or  misunderstanding." 

Supervisors:  "X  believe  it  would  be  helpful  to  give  a  more  detailed 

outline  of  what  is  Included  in  the  course." 

"Cannot  say,  since  X  have  not  seen  any  'advance  infor¬ 
mation.'  Z  knew  of  the  course  by  hearsay." 

"I  have  never  been  provided  anything  in  the  way  of 
ad\  sacs  information  on  what  the  course  is  other  than  a 
'Problem-Solving  and  Decision-Making  Course.'  To  make 
a  full  response  to  this  type  of  questionnaire,  should 
know  more  about  the  course." 

"What  is  meant  by  ' other \  advance  information?  I  don't 
know  what  was  provided." 
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"I  would  have  appreciated  Information  on  the  objectives  of 
the  course,  its  content,  how  it  was  to  be  conducted,  and 
the  qualifications  of  the  instructors.  Had  I  known  all 
this,  I  might  not  have  recommended  this  employee  take  it." 
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One «t ion  Pre-7:  WHO  PARTICIPATED  IN  DECIDING  THAT  YOU/THIS  EMPLOYEE 
WERE/WAS  TO  ATTEND? 


Level  2  -  Answers  that  explicitly  mention  the  employee  as  particlpati 
in  the  decision. 

Level  1^  -  Answers  which  indicate  definite  knowledge  of  participation 
others  than  employee,  but  do  not  explicitly  mention  the  employee. 

Level  0  -  Any  answer  which  indicates  mere  supposition,  lack  of  knowl . 
on  this  point,  or  no  participation  by  others. 


Exhibit  9 


Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Question  Pre-7 
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Exhibit  9  shows  very  close  agreement  between  supervisors  and  par 
cipants.  Eleven  percent  of  the  participants  were  unable  to  say  who  h 
taken  part  in  the  decision  that  they  should  receive  training.  All  of 
these  individuals,  in  answer  to  Pre-Question  4,  said  that  they  had  no 
discussed  the  course  with  their  supervisors. 

The  fact  that  the  participant  himself  was  not  explicitly  mention 
in  answer  to  Question  Pre-7  does  not  necessarily  mean  that  he  did  not 
take  part  in  the  decision  concerning  his  taking  the  course.  In  fact, 
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five  individuals  who  said  that  they  had  participated  In  the  decision  were 
not  mentioned  by  their  supervisors.  Two  more  were  mentioned  by  their 
supervisors  but  not  by  themselves. 
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Question  Pre-8:  OTHER  COMflfftt 


Levi  JL  -  If  eny  coaments  arp  giyen. 

Level  0  -  If  no  consents  ere  gjivm. 

Thirty-eight  percent  of  the  participants  and  25%  of  the  supervisors 
offered  further  comments.  No  tittle  these  stood  out  sharply  in  the  re¬ 
marks  of  either  group.  Participants  expressed  the  hope  that  the  course 
would  prove  profitable;  several  eAlarged  upon  benefits  that  they  hoped  to 
gain  from  it.  One  or  two  supettliOrs  expressed  similar  ideas;  others  com* 
plained  that  the  questionnaire  had  not  reached  them  earlier. 

The  results  obtained  from  the  two  pre- training  questionnaires  have 
implications  which  themselves  are  beat  stated  as  qvstlons.  Thus,  we  m a, 

ask: 


1.  Is  there  sufficient  communication  between  training  staff  and 
line  supervisors  concerning  the  content  and  objectives  of  available 
courses?  Is  the  supervisor  usually  provided  with  enough  Information  to 
enable  him  to  make  the  best  choices  among  courses  and  among  potential 
trainees? 


2.  If  a  supervisor  is  unable  to  documsnt  specific  training  needs  or 
to  specify  ways  in  which  he  expects  his  employees  to  apply  the  proposed 
training,  on  what  basis  does  he  select  trainees?  Is  too  heavy  reliance 
placed  upon  the  "quota?  method  of  assignment?  Can  some  better  methods  of 
selecting  participants  be  found?  Is  the  ms  re  hope  that  an  employee  will 
be  "helped"  by  a  course,  about  which  the  supervisor  knows  little  more  than 
its  title,  sufficient  Justification  for  the  cost  of  the  training? 

3.  Is  the  employee  himself  generally  slimed  a  voice  in  the  selec¬ 
tion  of  training?  Do  supervisors  sufficiently  discuss  the  proposed  train¬ 
ing  with  their  employees? 
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Post -Training 

The  post-training  questionnaire  for  supervisors  contains  three  ques¬ 
tions  including  "other  consents."  The  participant's  questionnaire  con¬ 
tains  five,  two  of  which  have  two  parts  each,  making  seven  questions  in 
all.  In  the  following  presentation,  results  for  both  supervisors  and 
participants  will  be  discussed  together  for  three  of  the  questions.  The 
remaining  ones  apply  to  participants  alone. 


24 


Question  post-1:  IN  WHAT  WAYS,  IF  ANY,  HAVE  YOUR  IDEAS  CHANGED  AS  TO 
THE  OBJECTIVES  OF  THE  COURSE?  (Supervisors  end 
Participants) 


Level  2  -  Responses  enumerating  specific  instance(s)  of  new  or  enlarged 
ideas  of  objectives. 

"tty  ideas  have  enlarged  rather  than  changed.  In  addition  to  the 
four  objectives  I  gave  before,  I  should  now  add:  (1)  to  learn 
how  to  assign  priorities  to  problems  awaiting  solution;  (2)  to 
know  how  to  establish  controls  in  order  that  the  decision  may  be 
carried  out  in  the  proper  manner;  (3)  to  be  able  to  develop  alter¬ 
natives  as  an  adjunct  to  decision-making." 

"Yes.  I  originally  thought  of  the  course  in  terra  of  decision¬ 
making  only.  Now  I  consider  that  the  objectives  are:  (1)  to 
consciously  and  systematically  analyse  each  problem  situation; 

(2)  to  assess  causes  of  problems  and  systematically  arrive  at 
decisions." 

Level  1  -  Statement  deals  in  generalities:  Ideas  have  change!,  but 
respondent  does  not  specify  particular  objectives  concerning  which  his 
thinking  has  undergone  change. 

"I  did  not  have  any  specific  ideas  as  to  the  objectives  of  the 
course  before  attending  it.  I  now  understand  the  objectives  and 
consider  them  worthy  and  attainable,  at  least  in  part.  The  course 
was  more  concrete  end  applicable,  with  less  pure  theory  than  I 
would  have  spec ted." 

"X  believe  that  the  rules  and  principles  taught  in  this  course  are 
not  as  specific  as  X  had  expected;  that  is,  the  course  may  improve 
abilities  along  these  lines,  but  will  not  replace  ability." 

Level  £  -  Responses  indicating  that  ideas  have  not  changed. 
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Exhibit  10 

Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Ouestion  Post-1 
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The  responses  of  the  two  groups  are  compared  in  Exhibit  10.  Of 
the  supervisors,  21**  reported  changes  in  their  ideas  as  to  objectives 
of  the  course  (Levels  1  and  2  combined),  whereas  58%  of  the  partici¬ 
pants  reported  change.  This  is  to  be  expected  since  the  participants 
were  the  ones  who  had  actually  bean  exposed  to  the  training.  If  a 
supervisor  had  not  discussed  the  course  with  his  employee  since  the 
latter* a  return,  he  naturally  would  have  had  no  opportunity  to  change 
his  ldean. 

Ou  the  other  hard,  sons  individuals— both  participants  end  super¬ 
visors— who  reported  "no  change,"  explained  that  they  had  had  a  reason¬ 
ably  clear  idea  of  the  objectives  to  begin  with.  A  few  supervisors  had 
gained  their  ideas  ft on  first-hand  axper lance,  having  praviously  taken 
the  course  thease'ves. 
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Question  font* 2a:  GIVE  EXAMPLES  OF  SITUATIONS  IN  YOUR  OHN  HOKE  IN  WHICH 
THE  CONCEPTS  AMD  METHODS  TAUGHT  IN  THIS  COURSE  WOULD 
BE  APPLICAB1E.  (Participants  only) 


Lsvsl  ^  -  Response  motions  apacl£lc  taak(a)  or  problan(s). 

"Carrying  out  instructions  of  superiors  when  (1)  superior  gives  no 
Instructions  as  to  usthod  of  jperatlon;  (2)  more  Information  is 

needed  in  cedar  ta  make  plans." 

"Getting  more  out  of  nestings:  How  to  ask  good  questions  to  obtain 
Information  needed." 

Level  JL  -  Reference  to  general  task  or  problem  area,  but  without  specific 
as amp las. 

"I  believe  the  concepts  and  principles  tsught  In  this  course  could 
he  need  effectively  in  analysing  policy  change  proposals  and  subse¬ 
quent  directive  issuances." 

"I  feel  it  Is  an  excellent  approach  to  identify  problems  relating 
to  the  nee  of  operating  forme." 

jLavel  0  -  Ho  situations  or  problems  mentioned. 

"IS  m  no A  In  research  the  concepts  and  methods  are  applicable  in 
pcdblam  analysis  phases  and  the  day-to-day  oparatlonal  aspects." 

"The  prteatplea  involved  ere  associated  in  just  about  avsry  nova  1 
wain  in  tin  area  of  policy  making  and  decisions  affecting  Padaral 
Covet  meant  end  public  interest  **•.  our  Service  programs." 

This  question  is  cue  of  ssvsrsl  that  were  introduced' following  a 
revision  of  tbs  poet-training  questionnaire;  consequently,  it  was  not 
asked  of  ell  participants.  It  Is  essentially  s  repetition  of  pre- 
tralalag  Question  3.  Changes  in  the  distribution  of  responses  obtained 
before  and  after  training,  therefore,  can  tell  ua  some  thing  as  to  the 
effectiveness  of  the  course,  since  the  ability  to  recognise  possible 
applications  is  one  criterion  of  learning  in  this  csss. 
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COMPARISON  OF  QUESTIONS  PRE-3  AND  QUESTION  POST-2* 


Forty-one  participant*  answered  both  questions.  Exhibit  11  shows 
the  percentage  distributions  of  their  "before"  and  "after1'  responses. 
The  shift  is  iapressive  and  highly  significant  statistically. 5  Thus, 
following  training,  46X  of  the  group  were  able  to  suggest  specific 
applications  in  their  regular  jobs  as  compared  with  only  26X  before 
training.  Similarly,  the  proportion  of  participants  who  were  unable 
to  visualize  any  useful  applications  of  the  course  was  cut  from  34X 
to  14X  following  training. 


Exhibit  11 


Percentage  Distributions  of  Responses  by  Participants 
to  Questions  Pre-3  and  Post  2a 

100  I - 1 
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■*A  two-tailed  sign  test  was  significant  at  the  02X  level  of 
confidence. 
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Question  Post -2b:  GIVE  EXAMPLES  OF  SITUATIONf  Of  WHICH  THEY  WOULD  NOT  BE 
APPLICABLE.  (Participants  inly) 


l£SSl  1  *  EMfrw*s  Want  Iona  One  or  more  situations. 

2  “  WgSFCOflant  tan  thinly  of  no  auch  situation,  or  Indicates  that 
concepts  « in#  method*  WOtild  be  applicable  In  alleituatione. 

In  their  responses  to  this  question,  42  participants  split  almost 
50-50.  Fifty- twb  percent  were  able  to  indicate  situations  In  which  they 
felt  the  method  of  pedblem  analysis  and  deciaiqo-aaking  taught  in  this 
course  would  apt  *e  Applicable.  The  remaining  were  unable  to  think 
of  any  such  situation- -largely  because,  as  many  respondent a  explained, 
they  felt  that  the  methods  would  be  applicable,  at  least  to  some  degree, 
in  any  situation.  It  is  interesting  to  note  in  passing  that,  of  22  indi¬ 
viduals  who  mentioned  specific  situations  or  type*  of  problem,  9  singled 
out  personnel  WQmk  AS  a  field  in  which  the  Kepner-Tregoe  approach  would 
not  apply. 


9 
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Question  Post -3a:  CAN  YOU  RECALL  ANY  SITUATION  DURING  THE  PAST  SIX  MONTHS 
OR  SO  THAT  YOU  MIGHT  HAVE  HANDLED  DIFFERENTLY  IF  YOU 
HAD  ALREADY  TAKEN  THE  COURSE?  (Perticipante  only) 


Level  2  -  Responses  which  mention  specific  sltustlon(s) . 

Level  i  -  Affirmative  answer,  but  no  situation  mentioned.  Note,  however, 
diet  situation(s)  may  be  mentioned  in  reply  to  Question  3b,  in  which  case 
a  score  of  2  is  assigned  to  3a. 

Level  0  -  Negative  answer. 

Fifty-two  participants  replied  to  this  question,  the  percentages  of 
Level  2,  1,  and  0  responses  being  38%,  24%,  and  38%  respectively.  Whon 
categories  2  and  1  are  combined,  It  appears  that  62%  recalled  situations 
which  they  might  have  handled  differently  if  they  had  taken  the  course 
sooner 
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C0KPA1ISCN  OF  POST -QUEST IONS  2a  AND  3a 


A  comparison  of  participants 1  responses  to  Questions  2a  and  3a 
ylalds  sons  Interesting  findings.  Levels  1  and  2  have  been  combined  to 
produce  Exhibit  12  as  shown  below. 


Exhibit  12 

Distribution  of  Is s ponses  to  Questions  2a  and  3a 
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From  Exhibit  12  it  appears  that,  following  training: 

301  of  the  participants  were  able  both  to  specify  applications  of 
tha  training  and  to  cite  situations  which  they  would  have  handled 
differently; 

36X  indicated  possible  applications  but  were  unable  to  recall  any 
situations  that  they  would  have  handled  differently- -at  least,  not 
during  the  past  six  months; 

7%  were  unable  to  indicate  applications,  but  could  recall  situa¬ 
tions  that  they  would  have  handled  differently  (clearly  an 
inconsistency) ; 

7X  were  unable  to  indicate  either  applications  or  situations  of 
tha  type  mentioned. 

Thus,  accepting  the  questionnaire  information  at  its  face  value,  we 
find. unequivocal  evidence  of  payoff  in  half  the  participants.  The  ques¬ 
tion  is  "Is  this  a  satisfactory  return  on  the  training  investment?"  A 
more  couplets  basis  of  Judgment,  of  course,  must  await  tha  returns  from 
the  six-month  follow-up.  It  may  wsll  be  asked,  however,  whether  more 
discriminating  selection  end  preparation  of  participants  night  not  improve 
the  picture,  even  at  this  stage. 
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Question  Post-Ju:  If  i'££  .  IN  '**aT  «4Yt  DO  ’.•-o  1,!..  *  Vi  J  od'iC*/  V  ..  .NT 
Have,*  BEEN  DIFFERENT?  (  J»  E  SPEJiFo.  ..XAi<t’.£S. 


Level  2  -  Specific  example  of  way(s)  in  which  outcome  might  have  been 
changed  for  better. 

"The  three  alternate  methods  of  reporting  performance  and  claiming 
reimbursement  in  the  Special  X  Program  might  have  been  reduced  to 
one  standard  method  " 

"In  the  area  of  procurement,  we  had  a  crash  program  in  which  we 
should  have  anticipated  places  where  it  could  go  wrong  and  have 
been  ready  with  back-up  alternatives,  and  have  established  effec¬ 
tive  controls." 

Level  2  "  General  statement--specific  outcome  not  detailed. 

"Results  may  have  been  approximately  the  same,  but  project  propo¬ 
sals  would  have  been  better  prepared  and  easier  to  follow  up.” 

"Now  that  the  course  is  over,  I  believe  I  would  have  felt  greater 
assurance  of  the  'reliability'  of  the  decisions.  The  steps  leading 
up  to  the  final  decision  would  have  been  sude  more  'consciously'." 

Level  0  -  No  explanation  as  to  how  outcome  would  have  been  different. 

Thirty-two  participants  responded  at  either  Level  1  or  Level  2  on 
Question  Post-3b;  thereby  Indicating  soma  situation  or  problem  ares  that 
they  might  have  handled  differently  after  having  completed  the  Kepner- 
Tregoe  course.  Of  these  32  individuals: 

SOI  were  able  to  specify  in  some  detsll  the  kind  of  Improved 
outcome  that  might  have  be.»n  expected; 

38Z  were  able  to  suggest  ths  probabls  out cons  in  more  general 

terms; 

12%  failed  to  haxard  a  guess  on  this  point. 
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Question  Post-4:  WHAT  PLANS  DO  YOU  HAVE  FOR  DISCUSSING  THIS  COURSE  WITH 
YOUR  SUPERVISOR?  (Psrticlpsnts) 

IN  WHAT  WAYS  DO  YOU  PLAN  TO  FOLLOW  UP  ON  THIS  EMPLOYEE'S 
USE  OF  WHAT  HE  LEARNED  IN  THE  COURSE?  (Supervisors' 
Question  2) 


Level  2  -  Statement  mentions  content  of  discussion  as  planned  or  already 
held,  or  indicates  presentation  to  staff  meeting. 

"Have  touched  on  K-T  program  at  PM  staff  meeting  last  week.  Have 
spent  45  minutes  briefing  X,  Service  Training  Officer,  on  my  reac¬ 
tions  to  program.  I  expect  to  visit  in  detail  with  my  supervisor, 
Director  Y,  in  next  week  or  two." 

"He  has  been  urged  to  utilise  all  he  learned  about  decision-making 
in  the  K-T  Course  in  dealing  with  employee  problems  as  well  as 
deciding  paralssable  tolerances  for  map  construction  accuracy.  We 
shall  require  that  his  recommendations  for  action  on  employee  pro¬ 
motions,  transfers  and  welfare  as  well  as  for  mapping  tolerances 
show  evidence  that  he  and  his  staff  have  arrived  at  the  decision  by 
objective  and  discerning  analysis.  He  will  be  required  to  analyze 
his  decisions  for  the  Division  Staff." 

Level  l  -  General  statement  with  no  specific  details  or  examples  given. 

"I  plan  to  discuss  the  course  with  him,  perhaps  at  some  length,  as 
office  or  off-duty  time  permit." 

"I  plan  to  work  closely  with  him  in  applying  these  procedures  to 
our  management  problems  whenever  such  procedures  are  suitable  and 
applicable." 

Level  0  -  No  discussion  planned. 

"I  do  not  intend  to  develop  e  plan  for  follow-up  on  the  employee. 

I  do  intend  to  observe  mors  closely  the  quality  of  the  employee's 
completed  staff  assignments." 

"Will  not  sea  him  before  January  4  and  have  not  discussed  course 
with  him  since  it  was  completed." 
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Exhibit  13 

Percentage  Distributions  of  Responses  by  Participants 
and  Supervisors  to  Questions  Post-4  and  Post-2 

100  I - 1 


As  Exhibit  13  shows,  participants  and  supervisors  showed  good 
general  agraaaant  in  respect  to  this  question,  and  nost  of  then  indi¬ 
cated  soon  plans  to  follow-up. 
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Question  ft>at-$;  Ollllft  COMBNTS  (Participants  and  Suparvipors) 


Level  _i  -  If  any  cooaenta  are  given. 

Level  0  -  If  no  cooaenta  are  given. 

Of  participants,  69%  made  comments;  9i  supervisors,  44%.  It  la 
perhaps  worth  noting  that  both  of  these  percentages  are  higher  than  the 
corresponding  figures  for  the  pre-cralnlng  questionnaires  (38%  and  25% 
respectively),  although,  as  before,  a  smaller  proportion  of  supervisors 
ttss  of  participants  had  coonantt  to  offer. 

Eleven  supervisors  offered  consents.  Vine  of  these  centered  upon  a 
Single  theme— their  subordinates  felt  the  course  to  be  amtreme ly  worth¬ 
while.  Two  raspdndsnts  had  acquired  fairly  specific  ideas  as  to  the  value 
at  tin  training.  Others  ware  more  vague  IS,  for  example,  the  mystified 
supervisor  who  said: 

MI  talftsd  vary  briefly  on  the  telepMhs  trfttfc  the  participant  about 
the  course.  He  said  that  it  was  wortbefcilm,  but  did  not  specify 
in  what  way  it  was  worthwhila.  I  have  now  talked  with  three  men 
gho  have  had  tha  eouria  and  thay  have  all  said  that  it  was  worth- 
likila,  bUt  did  not  tall  ms  why.  I'm  intftgmd  by  this  'secrecy'." 

The  participants,  as  might  bs  expected,  «srs  mush  more  varied  in 
their  Bnsisnte .  Among  the  points  most  frigwily  hsmtioned  were  the 
fgUdvds^e 

CA%  pralssd  thslr  instructor; 

fit  fait  that  ths  csurss  should  t»  jphNh  ht  an  earlier  stage 
in  the  individual's  carsar  devalqbnsrttl 

U%  felt  that  nor*  t%me  should  ba  afthpetad  9#  tha  training; 

tt)t  sgraad  that  tha  dac is ion -making  part  of  tha  course  was 
lass  sffsctlvs  chan  ths  part  dsadimp  mix*,  problem  analysis. 


V.  Discussion 


Results  presented  in  the  last  chapter  provide  an  incomplete  valu  ¬ 
ation  of  the  Kepner-Tregoe  course.  The  real  test  will  come  with  t'rs 
follow-up--this  will  give  some  indication  of  the  extent  to  which  j.~rti- 
cipants  have  put  the  concepts  and  methods  to  use  in  their  jobs.  / t  the 
present  stage,  this  much  can  be  said:  If  a  Level  1  or  Level  2  an.^er 
to  both  Post-Questions  2a  and  3a  be  accepted  as  a  criterion  of  success, 
then  exactly  half  of  the  42  participants  for  whom  complete  data  were 
available  successfully  completed  the  course.  Six  of  the  42  failed  to 
meet  this  criterion,  and  15  fell  in  the  doubtful  category  (Exhibit  12). 

Is  this  a  good  showing? 

On  the  other  hand,  our  findings  up  to  this  point  raise  some  inter¬ 
esting  questions  regarding  the  specific  requirements  of  a  sound  training 
program  as  described  in  Section  II  of  this  report. 

Consider  first  a  question  of  communication.  Twenty-seven  percent 
of  the  supervisors  and  38£  of  the  participants  felt  that  the  advance 
information  provided  them  concerning  the  course  was  insufficient.  How, 
then,  could  they  arrive  at  a  sound  decision  concerning  the  objectives  of 
the  course  or  its  value?  Fortunately,  in  this  case  the  course  happened 
to  be  a  good  one;  but  will  this  always  be  true?  Would  it  not  strengthen 
the  overall  training  effort  if  course  announcements  were  screened  for 
adequacy  of  information  before  they  were  passed  along  to  agency  training 
officers  and  supervisors?  A  brief  memorandum  calling  attention  to  this 
point  might  well  be  appended  to  the  announcement;  on  the  baois  o f  the 
adequacy  of  information  the  course  might  be  recommended  or  not  recom¬ 
mended  for  consideration.  Would  this  be  desirable? 

Another  facet  of  the  communication  problem  appears  in  the  fact  that 
in  approximately  one-half  of  the  cases  no  discussion  between  supervisor 
and  employee  occurred  prior  to  the  latter's  participation  in  the  Kepner- 
Tregoe  course.  Of  course--as  our  own  data  show — prior  discussion  with 
one's  supervisor  is  not  an  essential  condition  of  learning.  Nevertheless, 
some  communication  between  employee  and  supervisor  would  seem  to  be  desir¬ 
able  from  the  standpoint  of  attention  to  training  needs,  a  wise  selection 
of  training  programs,  and  the  integration  of  training  with  operations. 

Is  it  Indicative  of  cooperative  planning  by  supervisor  and  employee  when 
a  participant  writes,  "I  have  no  idea  what  my  supervisor's  ideas  are"? 

« 

It  was  suggested  in  the  second  chapter  that  the  systematic  use  of 
an  evaluation  procedure  such  as  this,  should  lead  eventually  to  better 
results  from  the  training  effort.  The  evaluation  technique  described  in 
this  report  is  now  being  extended  to  two  other  training  programs:  Seminars 
in  Executive  Development  (SED)  and  the  newly  inaugurated  Seminar  in  Middle 
Management  (SIMM),  both  of  which  are  Interagency  programs.  In  the  case  of 
SIMI-,  the  proposed  evaluation  is  mentioned  in  the  brochure  announcing  the 
program. 
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Although  it  is  desirable,  it  is  not  alvays  necessary  to  make  as 
detailed  an  analysis  of  results  as  was  done  in  this  case.  The  full-scale 
treatment  of  evaluation  data  was  offered  here  to  show  how  it  can  be  done. 
But  careful  examination  of  the  questionnaire  responses  alone,  without 
formal  content  analysis  or  coding,  can  yield  significant  information 
about  the  effectiveness  of  the  overall  training  effort.  The  main  aim  is 
to  provide  for  specific  plans  oriented  to  individual  development  needs 
measured  against  defined  management  objectives,  to  provide  fur  shared 
understanding  r<f  training  purposes,  to  focus  on  specific  behavioral  ef¬ 
fects  of  training  and  upon  the  use  of  such  performance  evidence  in  evalu¬ 
ating  trafc»ing--in  short,  to  create  situational  conditions  facilitating 
the  effective  utilization  of  training  received. 

To  tiwlize  the  potential  benefits  of  evaluation,  however,  it  is  nec¬ 
essary  that  the  procedure  be  carried  out  Methodically  and  on  a  continuing 
basis.  This  means,  among  other  things,  insisting  upon  complete  returns 
from  both  participants  and  their  supervisors  on  all  questionnaires .  Un¬ 
less  thM  requirement  is  observed,  the  entire  effort  becomes  pointless. 

If  mandgsaient  its. If  treats  the  evaluation  casually,  others  will  do  30 
too.  the  result  can  be  an  altogether  unwarranted  complacency  regarding 
the  effectiveness  of  the  entire  training  an«!  management  development 
progrdg. 


'  A'  P  ?:  k  k  i>  i  x  k 
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TRAINING  RESEARCH  QUESTIONNAIRE 

INSTRUCTIONS  FOR  PARTICIPANTS 

A  considerable  number  of  people  in  the  Department  have  taken  or 
soon  will  participate  In  the  Kepner-Tregoe  training  program  in 
Management  Problem  Analysis  and  Decision-Making.  In  order  to  find  out 
v**t  values  this  training  may  have  for  our  employees,  it  is  necessary 
to  obtain  some  information  before  as  well  as  after  you  have  attended 
cbe  course. 

In  answering  the  accompanying  questions,  please  be  as  specific  as 
poeslb .  When  you  have  completed  the  course,  you  may  be  asked  some 
more  questions  about  it.  We  also  plan  to  follow  up  from  time  to  time 
.n  the  future  to  check  on  long-range  benefits  from  the  program. 

the  most  valuable  information  is  your  frank  reactions  based  upon 
your  personal  expectations,  needs,  experience  end  applications.  So, 
work  Independently- -do  not  consult  anyone  else  in  answering  this 
quest leauelre.  Peel  free  to  react  favorably  or  unfavorably. 

M  bleak  sheets  are  provided  for  your  responses  (fe<*l  free  to 
lisa  additional  sheets  it  you  need  them;-.  Pleaje  number  your  answers 
to  correspond  with  the  questions. 


Hunk  you. 
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Quest  ions  for  Participants  Prioi  to  Training 

1.  What  do  you  think  are  the  specific  objectives  of  the  course? 

2.  How  does  this  seem  to  fit  into  your  own  career  plans? 

3.  On  the  basis  of  information  available  to  you  now,  in  what  particular 
ways  do  you  expect  to  apply  this  training  in  your  future  work?  Give 
examples. 

4.  Have  you  discussed  points  1,  2,  and  3  with  your  supervisor?  If  Yes, 
do  you  feel  that  he  shares  the  same  expectations  as  you  do?  If  not, 
how  do  your  expectations  and  his  differ  as  you  see  it? 

5.  Looking  back  over  youx  work  during  the  past  six  r  mths  or  so,  has 
anything  happened  to  bring  home  to  you  a  need  for  this  kind  of 
training?  Give  actual  examples. 

6.  Is  there  any  other  advance  information  you  think  should  have  been 
provided  about  the  ccursa? 

7.  Who  participated  in  deciding  that  you  were  to  attend? 


8. 


Other  roessencs 
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TRAINING  RESEARCH  QUESTIONNAIRE 

INSTRUCTIONS  FOR  SUPERVISORS 

A  considerable  number  of  people  In  the  Department  have  taken  or  soon 
will  participate  in  the  Kepner-Tregoe  training  program  in  Management 
Problem  Analysis  and  Decision-Making.  In  order  to  find  out  what  values 
this  training  may  have  for  our  employees,  it  is  necessary  to  obtain  some 
information  before  as  well  as  after  they  have  attended  the  course. 

Please  answer  the  accompanying  questions  for  each  of  your  employees 
who  is  about  to  take  the  course.  In  giving  your  answers,  please  be  as 
specific  as  possible.  When  your  employee(s)  have  completed  the  course, 
you  may  be  asked  some  more  questions.  We  also  plan  to  follow  up  from 
time  to  time  In  the  future  to  check  on  any  long-range  benefits  from  the 
program. 

The  most  valuable  Information  is  your  frank  reactions  based  upon 
your  personal  experience  with  your  employees,  including  your  needs  and 
expectations  concerning  them.  So  work  Independently --do  not  consult 
anyone  else  in  answering  this  questionnaire.  Feel  free  to  react 
favorably  or  unfavorably. 

Two  blank  sheets  are  provided  for  your  responses  (feel  free  to  use 
additional  sheets  if  yju  need  them).  Please  number  your  answers  to 
correspond  with  the  questions. 


Thank  you. 
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Questions  for  Participants'  Supervisors  Prior  to  Traininn 

1.  What  do  you  think  are  tue  specific  objectives  of  the  course? 

2.  How  does  this  fit  into  your  Agency  Training  and  Development  Program 
for  this  participant? 

3.  On  the  basis  of  information  available  to  you  now,  in  vhar  particular 
ways  do  you  expect  this  employee  to  apply  the  training  in  his  future 
work?  Give  examples. 

4.  Have  you  discussed  points  1,  2,  and  3  with  the  employee?  If  Yes, 
do  you  feel  that  he  shares  the  same  expectations  as  you  do?  If 
not,  how  do  your  expectations  and  his  differ  as  you  see  it? 

5.  Looking  back  over  his  work  during  the  past  six  months  or  so,  has 
anything  happened  to  bring  hone  to  you  his  need  for  this  kind  of 
training?  Give  actual  examples. 

6.  Is  there  any  other  advance  information  you  think  should  have  been 
provided  about  the  course? 

7.  Who  participated  in  tfacld  ng  that  this  employee  was  to  attend? 

8.  Other  comments. 
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TRAINING  RESEARCH  QUESTIONNAIRE 

INSTRUCTIONS  FOR  PARTICIPANTS 

Now  that  you  have  completed  the  Kepner-Ttegoe  court e  in  "Management 
Problem  Analysis  and  Decision-Making,"  we  would  like  to  have  your  frank 
reactions  on  certain  points.  As  explained  in  the  pre-training  question 
naira  which  you  filled  out  earlier,  we  also  plan  to  follow  up  in  the 
future  in  order  to  check  on  long-term  effects  of  the  program. 

Feel  free  to  react  favorably  or  unfavorably  or  to  agree  or  disagree 
with  anyone  else's  point  of  view.  What  we  want  are  your  reactions, 
based  upon  your  own  personal  experience.  In  answering  these  questions, 
please  be  as  specific  as  possible. 

As  before,  two  blank  sheets  ere  provided  for  your  responses  (do  not 
hesitate  to  u^e  additional  sheets  if  you  need  them).  Please  number  your 
answers  to  correspond  with  the  questions. 

Thank  you  again. 
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Que.  tlons  for  Participants  after  Training 

1.  In  what  ways,  if  any,  have  your  ideas  changed  as  to  the  objectives 
of  the  course? 

2.  a.  Give  examples  of  situations  in  your  own  work  in  which  the  con¬ 

cepts  and  methods  taught  in  this  course  would  be  applicable, 
b.  Give  examples  of  situations  in  which  they  would  not  be  applicable. 

3.  a.  Can  you  recall  any  situations  during  the  past  six  months  or  so 

that  you  might  have  handled  differently  if  you  had  already  taken 
the  course? 

b.  If  yes,  in  what  ways  do  you  think  that  the  outcome  might  have 
been  different?  Give  specific  examples. 

4.  What  plans  do  you  have  for  dlscufiing  this  course  with  your  supervisor? 


5. 


Other  comments. 
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TRAINING  RESEARCH  QUESTIONNAIRE 

INSTRUCTIONS  FOR  SUPERVISORS 

Now  that  your  employee  has  completed  the  Kepner-Tregoe  course  in 
'‘Management  Problem  Analysis  and  Decision-Making,"  we  would  like  you  to 
answer  two  more  questions.  At.  explained  in  the  pre-training  questionnaire 
which  you  filled  out,  we  also  plan  to  follow  up  in  the  future  in  order  to 
check  on  long-term  effects  of  the  program. 

Please  answer  the  questions  below  for  each  employee  who  has  just 
finished  the  course.  You  probably  will  want  to  talk  to  the  employee 
before  you  start.  However  feel  free  to  react  favorably  or  unfavorably 
or  to  agree  or  disagree  with  anyone  else:s  point  of  view.  What  we  want 
are  your  reactions,  based  upon  your  own  experience  and  observations.  In 
giving  your  answers,  please  be  as  specific  as  possible. 

As  before,  two  blank  sheet J  are  provided  for  your  responses  (do  not 
hesitate  to  vise  additional  sheets  if  you  need  them).  Please  number  your 
answers  to  correspond  with  the  questions 
Thvuik  you  again. 

Questions 

1.  in  what  ways,  if  any,  have  your  Ideas  changed  as  to  the  objectives 
of  the  course? 

2.  In  what  ways  do  you  plan  to  follow  up  on  this  employee's  use  of 
what  he  learned  in  the  course? 

3.  Other  comments. 


